{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "coding_nn_with_bp_from_scratch.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPsr8gXE+bXcM7fd374EDha",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SummerLife/EmbeddedSystem/blob/master/MachineLearning/project/08_code_nn_from_scratch/coding_nn_with_bp_from_scratch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0AJK-lokeNv",
        "colab_type": "text"
      },
      "source": [
        "# Coding a Neural Network with Backpropagation from scratch\n",
        "\n",
        "- Initialize Network\n",
        "- Forward Propagate\n",
        "- Back Propagate Error\n",
        "- Train Network\n",
        "- Predict\n",
        "- Seeds Dataset Case Study"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "trbkkR6Gk4OW",
        "colab_type": "text"
      },
      "source": [
        "## 1. Initialize Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RyiTEZ_RgbTu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "4bd4a928-0d01-41ab-f5ee-4a6148664a74"
      },
      "source": [
        "from random import seed\n",
        "from random import random\n",
        "\n",
        "# Initialize a network\n",
        "def initialize_network(n_inputs, n_hidden, n_outputs):\n",
        "    network = list()\n",
        "    hidden_layer = [{\"weights\": [random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]\n",
        "    network.append(hidden_layer)\n",
        "    output_layer = [{\"weights\": [random() for i in range(n_hidden + 1)]} for i in range(n_outputs)]\n",
        "    network.append(output_layer)\n",
        "    return network\n",
        "\n",
        "seed(1)\n",
        "network = initialize_network(2,1,2)\n",
        "for layer in network:\n",
        "    print(layer)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[{'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}]\n",
            "[{'weights': [0.2550690257394217, 0.49543508709194095]}, {'weights': [0.4494910647887381, 0.651592972722763]}]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qchtDuxdnSzY",
        "colab_type": "text"
      },
      "source": [
        "## 2. Forward Propagate\n",
        "\n",
        "We can break forward propagation down into three parts:\n",
        "\n",
        "1. Neuron Activation\n",
        "2. Neuron Transfer\n",
        "3. Forward Propagation\n",
        "\n",
        "### 2.1 Neuron Activation\n",
        "\n",
        "`activation = sum(weight_i * input_i) + bias`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b6d1tUSbnzZ0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate neuron activation for an input\n",
        "def activate(weights, inputs):\n",
        "    activation = weights[-1]\n",
        "    for i in range(len(weights) - 1):\n",
        "        activation += weights[i] * inputs[i]\n",
        "    return activation"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "miR21ajCpbi-",
        "colab_type": "text"
      },
      "source": [
        "### 2.2 Neuron Transfer\n",
        "\n",
        "Once a neuron is activated, we need to transfer the activation to see what the neuron output actually is.\n",
        "\n",
        "The sigmoid activation function looks like an S shape, it's also called the logistic function. It can take any input value and produce an number between 0 and 1 on an S-curve. It is also a function of which we can easily calculate the derivatice(slope) that we will need later when backpropagating error.\n",
        "\n",
        "We can transfer an activation function using the sigmoed function as follows:\n",
        "\n",
        "`output = 1 / (1 + e^(-activation))`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "foOJ9XOrpQK1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Transfer neuron activation\n",
        "from math import exp\n",
        "\n",
        "def transfer(activation):\n",
        "    return 1.0 / (1.0 + exp(-activation))"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0gmPYrnCrxbW",
        "colab_type": "text"
      },
      "source": [
        "### 2.3 Forward Propagation\n",
        "\n",
        "Forward propagating an input is straightforward.\n",
        "\n",
        "We work through each layer of our network calculating the outputs for each neuron. All of the outputs from on layer become in puts to the neurons on the next layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lEdCUDsXrKTN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Forward propagate input to a network output\n",
        "def forward_propagate(network, row):\n",
        "    inputs = row\n",
        "    for layer in network:\n",
        "        new_inputs = []\n",
        "        for neuron in layer:\n",
        "            activation = activate(neuron['weights'], inputs)\n",
        "            neuron['output'] = transfer(activation)\n",
        "            new_inputs.append(neuron['output'])\n",
        "        inputs = new_inputs\n",
        "    return inputs"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_fmllGPZuovP",
        "colab_type": "text"
      },
      "source": [
        "### 2.4 Test out the forward propagation of out network\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9bdgPgSufCA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2f887050-dcef-4c1e-bf69-fd059961e5ad"
      },
      "source": [
        "row = [1, 0, None]\n",
        "output = forward_propagate(network, row)\n",
        "print(output)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0.6629970129852887, 0.7253160725279748]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q100NQ2SxIt_",
        "colab_type": "text"
      },
      "source": [
        "## 3. Back Propagate Error\n",
        "\n",
        "Error is calculated between the expected outputs and the outputs forward propagated from the network. These errors are then propagated backward through the netwoek from the output layer to the hidden layer, assigning blame for the error and updating weights as they go.\n",
        "\n",
        "1. Transfer Derivative\n",
        "2. Error Backpropagation\n",
        "\n",
        "### 3.1 Transfer Derivative\n",
        "\n",
        "We are using the sigmoid transfer function, the derivative of which can be calculated as follows:\n",
        "\n",
        "```\n",
        "derivative = output * (1.0 - output)\n",
        "```\n",
        "\n",
        "Using function named `transfer_derivative()` to implements this equation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6fJTj3nlvFqq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate the derivative of an neuron output\n",
        "def transfer_derivative(output):\n",
        "    return output * (1.0 - output)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZBM-iG38nz3",
        "colab_type": "text"
      },
      "source": [
        "### 3.2 Error Backpropagation\n",
        "\n",
        "The error for a given neuron can be calculated as follows:\n",
        "\n",
        "`error = (expected - output) * transfer_derivative(output)`\n",
        "\n",
        "\n",
        "The error signal for a neuron in the hidden layer is calculated as the weighted error of each neuron in the output layer. Think of the error traveling back along the weights of the output layer to the neurons in the hidden layer.\n",
        "\n",
        "The back-propagated error signal is accumulated and then used to determine the error for the neuron in the hidden layer, as follows:\n",
        "\n",
        "`error = (weight_k * error_j) * transfer_derivative(output)`\n",
        "\n",
        "Where error_j is the error signal from the jth neuron in the output layer, weight_k is the weight that connects the kth neuron to the current neuron and output is the output for the current neuron.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qqVpngAfpuBM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Backpropagate error and store in neurons\n",
        "def backward_propagate_error(network, expected):\n",
        "    for i in reversed(range(len(network))):\n",
        "        layer = network[i]\n",
        "        errors = list()\n",
        "\n",
        "        if i != len(network) - 1:\n",
        "            for j in range(len(layer)):\n",
        "                error = 0.0\n",
        "                for neuron in network[i + 1]:\n",
        "                    error += (neuron['weights'][j] * neuron['delta'])\n",
        "                    errors.append(error)\n",
        "        else:\n",
        "            # calculate the error signal from the neuron in the output layer\n",
        "            for j in range(len(layer)):\n",
        "                neuron = layer[j]\n",
        "                errors.append(expected[j] - neuron['output'])\n",
        "\n",
        "        for j in range(len(layer)):\n",
        "            neuron = layer[j]\n",
        "            neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7zzFy8O3qg7d",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "fc99e416-35d0-4585-d6b7-dcea703c777c"
      },
      "source": [
        "# test backpropagation of error\n",
        "network = [[{'output': 0.7105668883115941, 'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}],\n",
        "          [{'output': 0.6213859615555266, 'weights': [0.2550690257394217, 0.49543508709194095]}, \n",
        "           {'output': 0.6573693455986976, 'weights': [0.4494910647887381, 0.651592972722763]}]]\n",
        "expected = [0, 1]\n",
        "\n",
        "backward_propagate_error(network, expected)\n",
        "\n",
        "for layer in network:\n",
        "    print(layer)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[{'output': 0.7105668883115941, 'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614], 'delta': -0.007668854370284511}]\n",
            "[{'output': 0.6213859615555266, 'weights': [0.2550690257394217, 0.49543508709194095], 'delta': -0.14619064683582808}, {'output': 0.6573693455986976, 'weights': [0.4494910647887381, 0.651592972722763], 'delta': 0.0771723774346327}]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmWIgl2FwNbl",
        "colab_type": "text"
      },
      "source": [
        "## 4. Train Network\n",
        "\n",
        "This part is broken down into two sectinons:\n",
        "\n",
        "1. Update Weights\n",
        "2. Tran Network\n",
        "\n",
        "### 4.1 update weights\n",
        "\n",
        "Once errors are calculated for each neuron in the network via the back propagation method above, they can be used to update weights.\n",
        "\n",
        "Network weights are updated as follows:\n",
        "\n",
        "`weight = weight + learning_rate * error * input`\n",
        "\n",
        "Where weight is a given weight, learning_rate is a parameter that you must specify, error is the error calculated by the backpropagation procedure for the neuron and input is the input value that caused the error.\n",
        "\n",
        "The same procedure can be used for updating the bias weight, except there is no input term, or input is the fixed value of 1.0.\n",
        "\n",
        "Remember that the input for the output layer is a collection of outputs from the hidden layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qF0nGg42tnqw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Update network weights with error\n",
        "def update_weights(network, row, l_rate):\n",
        "    for i in range(len(network)):\n",
        "        inputs = row[:-1]\n",
        "\n",
        "        if i != 0:\n",
        "            inputs = [neuron['output'] for neuron in network[i - 1]]\n",
        "\n",
        "        for neuron in network[i]:\n",
        "            for j in range(len(inputs)):\n",
        "                neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]\n",
        "            neuron['weights'][-1] += l_rate * neuron['delta']"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fgIn5DQN2B3-",
        "colab_type": "text"
      },
      "source": [
        "### 4.2 Train Network\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-iift9oo2IHl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Train a net work for a fixed number of epochs\n",
        "def train_network(network, train, l_rate, n_epoch, n_outputs):\n",
        "    for epoch in range(n_epoch):\n",
        "        sum_error = 0\n",
        "        for row in train:\n",
        "            outputs = forward_propagate(network, row)\n",
        "            expected = [0 for i in range(n_outputs)]\n",
        "            expected[row[-1]] = 1\n",
        "            sum_error += sum([(expected[i] - outputs[i])**2 for i in range(len(expected))])\n",
        "            backward_propagate_error(network, expected)\n",
        "            update_weights(network, row, l_rate)\n",
        "        print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qn--O-kn4lCo",
        "colab_type": "text"
      },
      "source": [
        "### 4.3 Test Network\n",
        "\n",
        "We will use 2 neurons in the hidden layer. It is binary classification problem (2 classes) so there will be two neurons in the output layer. The network will be trained for 20 epochs with learning rate of 0.5, which is high because we are training for so few iterations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "THBoU0Cr2j6h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 411
        },
        "outputId": "8ea72353-fdbd-4ebc-d194-71f154fdf788"
      },
      "source": [
        "from math import exp\n",
        "from random import seed\n",
        "from random import random\n",
        "\n",
        "# Test training backprop algorithm\n",
        "seed(1)\n",
        "\n",
        "dataset = [[2.7810836,2.550537003,0],\n",
        "           [1.465489372,2.362125076,0],\n",
        "           [3.396561688,4.400293529,0],\n",
        "           [1.38807019,1.850220317,0],\n",
        "           [3.06407232,3.005305973,0],\n",
        "           [7.627531214,2.759262235,1],\n",
        "           [5.332441248,2.088626775,1],\n",
        "           [6.922596716,1.77106367,1],\n",
        "           [8.675418651,-0.242068655,1],\n",
        "           [7.673756466,3.508563011,1]]\n",
        "\n",
        "n_inputs = len(dataset[0]) - 1\n",
        "n_outputs = len(set([row[-1] for row in dataset]))\n",
        "network = initialize_network(n_inputs, 2, n_outputs)\n",
        "train_network(network, dataset, 0.5, 20, n_outputs)\n",
        "for layer in network:\n",
        "\tprint(layer)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ">epoch=0, lrate=0.500, error=6.365\n",
            ">epoch=1, lrate=0.500, error=5.557\n",
            ">epoch=2, lrate=0.500, error=5.291\n",
            ">epoch=3, lrate=0.500, error=5.262\n",
            ">epoch=4, lrate=0.500, error=5.217\n",
            ">epoch=5, lrate=0.500, error=4.899\n",
            ">epoch=6, lrate=0.500, error=4.419\n",
            ">epoch=7, lrate=0.500, error=3.900\n",
            ">epoch=8, lrate=0.500, error=3.461\n",
            ">epoch=9, lrate=0.500, error=3.087\n",
            ">epoch=10, lrate=0.500, error=2.758\n",
            ">epoch=11, lrate=0.500, error=2.468\n",
            ">epoch=12, lrate=0.500, error=2.213\n",
            ">epoch=13, lrate=0.500, error=1.989\n",
            ">epoch=14, lrate=0.500, error=1.792\n",
            ">epoch=15, lrate=0.500, error=1.621\n",
            ">epoch=16, lrate=0.500, error=1.470\n",
            ">epoch=17, lrate=0.500, error=1.339\n",
            ">epoch=18, lrate=0.500, error=1.223\n",
            ">epoch=19, lrate=0.500, error=1.122\n",
            "[{'weights': [-0.9766426647918854, 1.0573043092399, 0.7999535671683315], 'output': 0.05429927062285241, 'delta': -0.0035328621774792703}, {'weights': [-1.2245133652927975, 1.4766900503308025, 0.7507113892487565], 'output': 0.03737569585208105, 'delta': -0.005989297622698788}]\n",
            "[{'weights': [1.4965066037208181, 1.770264295168642, -1.28526000789383], 'output': 0.24698288711606625, 'delta': -0.04593445543099784}, {'weights': [-1.8260068779176126, -1.1775229580602165, 1.1610216434075609], 'output': 0.7292895947013409, 'delta': 0.05344534875231567}]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JQQrAd_I6aMH",
        "colab_type": "text"
      },
      "source": [
        "## Predict\n",
        "\n",
        "We hace already seen how to forward-propagate an input pattern to get an output. This is all we need to do to make a prediction. We can use the output values themselves directly as the probability of a pattern belonging to each output class.\n",
        "\n",
        "It may be more useful to turn this output back into a crisp class prediction. We can do this by selecting the class value with the larger probability. This is also called the [arg max function](https://en.wikipedia.org/wiki/Arg_max).\n",
        "\n",
        "Below is a function named **predict()** that implements this procedure. It returns the index in the network output that has the largest probability. It assumes that class values have been converted to integers starting at 0."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ljI9NP_X5frD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make a prediction with a network\n",
        "def predict(network, row):\n",
        "    outputs = forward_propagate(network, row)\n",
        "    return outputs.index(max(outputs))"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fvm1lKGF8RF7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "628f2d77-4618-4276-b8f2-bee7bcf776ac"
      },
      "source": [
        "# Test making predictions with the network\n",
        "dataset = [[2.7810836,2.550537003,0],\n",
        "           [1.465489372,2.362125076,0],\n",
        "           [3.396561688,4.400293529,0],\n",
        "           [1.38807019,1.850220317,0],\n",
        "           [3.06407232,3.005305973,0],\n",
        "           [7.627531214,2.759262235,1],\n",
        "           [5.332441248,2.088626775,1],\n",
        "           [6.922596716,1.77106367,1],\n",
        "           [8.675418651,-0.242068655,1],\n",
        "           [7.673756466,3.508563011,1]]\n",
        "\n",
        "network = [[{'weights': [-1.482313569067226, 1.8308790073202204, 1.078381922048799]}, \n",
        "            {'weights': [0.23244990332399884, 0.3621998343835864, 0.40289821191094327]}],\n",
        "          [ {'weights': [2.5001872433501404, 0.7887233511355132, -1.1026649757805829]}, \n",
        "            {'weights': [-2.429350576245497, 0.8357651039198697, 1.0699217181280656]}]]\n",
        "\n",
        "for row in dataset:\n",
        "\tprediction = predict(network, row)\n",
        "\tprint('Expected=%d, Got=%d' % (row[-1], prediction))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Expected=0, Got=0\n",
            "Expected=0, Got=0\n",
            "Expected=0, Got=0\n",
            "Expected=0, Got=0\n",
            "Expected=0, Got=0\n",
            "Expected=1, Got=1\n",
            "Expected=1, Got=1\n",
            "Expected=1, Got=1\n",
            "Expected=1, Got=1\n",
            "Expected=1, Got=1\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}